Implement Library README Discovery, Backfilling, and Publishing (#242) by SurbhiAgarwal1 · Pull Request #380 · open-telemetry/opentelemetry-ecosystem-explorer

SurbhiAgarwal1 · 2026-05-07T02:41:50Z

Implementation Detail: Library README Support (#242)

This document provides a technical breakdown of the changes implemented to support library README markdown files in the OpenTelemetry Ecosystem Explorer.

1. Registry Discovery & Extraction

Component: java-instrumentation-watcher
File: inventory_manager.py

The InventoryManager was enhanced to handle the library_readmes/ directory within the versioned registry.

Regex Parsing: Filenames like apache-httpclient-4.3-a3bae406cfcf.md are parsed using r"^(.*)-([a-f0-9]+)\.md$" to separate the library name from its content hash.
O(1) Map: Added load_library_readme_map(version) which caches this relationship, allowing the database builder to quickly correlate instrumentations to their READMEs.
Resilience: The loader handles missing directories and malformed files gracefully, logging warnings without interrupting the pipeline.

2. Metadata Augmentation & Backfilling

Component: explorer-db-builder
Files: main.py, metadata_backfiller.py

We integrated README support into the existing content-addressed metadata system.

Hash Injection: The main.py orchestrator augments both libraries and custom instrumentation metadata with the markdown_hash during the initial inventory load.
Backfill Propagation: By adding markdown_hash to BACKFILLABLE_FIELDS, the system ensures that older release versions (where the README might be missing in the registry) correctly inherit the link from newer versions if the library name matches.
Transformation Stability: Verified that the new field is preserved across all format transformations (0.1 through 0.5).

3. Database Writing & Publishing

Component: explorer-db-builder
File: database_writer.py

The DatabaseWriter manages the physical transfer of assets to the explorer's public storage.

Storage Path: /public/data/javaagent/markdown/{library-name}-{hash}.md.
Deduplication: Implemented an existence check to prevent redundant writes of identical content across different versions or libraries.
Error Handling: Failures in markdown publishing are caught and logged, ensuring that database generation is never blocked by individual file errors.

4. Frontend Integration

Component: ecosystem-explorer
Files: src/types/javaagent.ts, src/lib/api/javaagent-data.ts

Prepared the React frontend for content rendering:

Type Definition: Added markdown_hash?: string to the InstrumentationData interface.
Lazy Loading: Implemented loadLibraryReadme(libraryName, markdownHash) to fetch markdown content from the public directory on demand.

5. Build System & DX Improvements

Component: Multiple Watchers
Files: __init__.py

Modified the initialization logic to handle PackageNotFoundError. This allows developers to run the builder and tests directly from source in uninstalled environments (common in dev/CI containers).

6. Verification Summary

Tests: 309/309 passed (pytest ecosystem-automation/).
Linting: All modified files formatted and verified with ruff.
E2E Validation: Confirmed that v2.26.1 correctly backfilled README links from v2.27.0 and that the public/data directory contains the correct assets.

…pen-telemetry#242) - Extend InventoryManager to discover and load library READMEs from registry - Augment instrumentation metadata with markdown_hash and enable backfilling - Implement markdown publishing to public data directory in DatabaseWriter - Add frontend types and API support for README lazy loading

netlify · 2026-05-07T02:41:55Z

✅ Deploy Preview for otel-ecosystem-explorer ready!

Name	Link
🔨 Latest commit	`4505b8e`
🔍 Latest deploy log	https://app.netlify.com/projects/otel-ecosystem-explorer/deploys/6a06c290f2de610008ee705e
😎 Deploy Preview	https://deploy-preview-380--otel-ecosystem-explorer.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

linux-foundation-easycla · 2026-05-07T02:41:56Z

The committers listed above are authorized under a signed CLA.

✅ login: SurbhiAgarwal1 / name: SurbhiAgarwal (6689ea7, 762d41f, cd43b9f)
✅ login: SurbhiAgarwal1 / name: SurbhiAgarwal1 (0e712c9, 186b011, 2732a05, 4863395, 556b3b0, 5b1a322, 6afc5cb, 74e232a, c9bb873)

Copilot

Pull request overview

Adds end-to-end support for per-library README markdown assets in the Java agent ecosystem: discovery from the registry, propagation/backfilling into instrumentation metadata, publishing to the explorer’s public data directory, and a frontend API hook to lazy-load the markdown.

Changes:

Java instrumentation watcher: discover and load library README markdown assets from library_readmes/.
Explorer DB builder: inject/backfill markdown_hash into library/custom items and publish markdown files to public/data/javaagent/markdown/.
Frontend: extend Javaagent instrumentation types with markdown_hash and add an API helper to fetch README markdown on demand.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
ecosystem-explorer/src/types/javaagent.ts	Adds `markdown_hash` to `InstrumentationData` for README association.
ecosystem-explorer/src/lib/api/javaagent-data.ts	Adds `loadLibraryReadme()` for lazy-loading published markdown assets.
ecosystem-automation/java-instrumentation-watcher/src/java_instrumentation_watcher/inventory_manager.py	Adds README map discovery + content loading + filename parsing for `library_readmes/`.
ecosystem-automation/java-instrumentation-watcher/src/java_instrumentation_watcher/init.py	Makes `__version__` resilient when package metadata isn’t installed.
ecosystem-automation/explorer-db-builder/src/explorer_db_builder/metadata_backfiller.py	Backfills `markdown_hash` across versions.
ecosystem-automation/explorer-db-builder/src/explorer_db_builder/main.py	Publishes READMEs and augments inventories with `markdown_hash` before backfill/write.
ecosystem-automation/explorer-db-builder/src/explorer_db_builder/database_writer.py	Writes markdown assets to a content-addressed public directory with dedup.
ecosystem-automation/explorer-db-builder/src/explorer_db_builder/init.py	Makes `__version__` resilient when package metadata isn’t installed.
ecosystem-automation/collector-watcher/src/collector_watcher/init.py	Makes `__version__` resilient when package metadata isn’t installed.

jaydeluca

looks great! tested things locally and everything looks good. Just a few suggestions from myself and copilot related to some of the logic

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 7 comments.

- Implement deterministic README selection in InventoryManager (mtime + lexicographical).\n- Tighten README filename regex to enforce 12-char hashes.\n- Use explicit None checks for README content publishing.\n- Update loadLibraryReadme to use fetchWithCache with status reporting.\n- Add unit and integration tests for README publishing and backfilling.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

jaydeluca

looks good, can you just resolve the merge conflict?

…ing tests

…ublic/data from markdown lint

…telemetry#242)

jaydeluca

@SurbhiAgarwal1 the changes look good. I pushed a change to remove the generated files from the PR because its best to have that added as part of an automation generated PR to keep the changes easier to review/go back to.

It looks like there might be some other conflicts with main / other files unrelated that are showing changes in the diff. Could you look into that? then this will be good to go, thanks

…onflicts

SurbhiAgarwal1 · 2026-05-15T08:15:34Z

Hi @jaydeluca , I have stabilized the build for this PR and resolved the regressions in the frontend test suite. Is their any other changes needed.

SurbhiAgarwal1 requested review from a team as code owners May 7, 2026 02:41

jaydeluca requested a review from Copilot May 7, 2026 17:59

Copilot started reviewing on behalf of jaydeluca May 7, 2026 18:00 View session

Copilot AI reviewed May 7, 2026

View reviewed changes

jaydeluca reviewed May 7, 2026

View reviewed changes

Comment thread ...utomation/java-instrumentation-watcher/src/java_instrumentation_watcher/inventory_manager.py Outdated

Comment thread ecosystem-explorer/src/lib/api/javaagent-data.ts Outdated

lucacavenaghi97 mentioned this pull request May 7, 2026

feat(db-builder): integrate Weaver for semconv compliance checking (#97) #382

Open

Merge branch 'main' into feat/242-library-readme-support

cd43b9f

jaydeluca mentioned this pull request May 8, 2026

Java Instrumentation Detail - Standalone Library page #243

Open

Merge branch 'main' into feat/242-library-readme-support

6689ea7

SurbhiAgarwal1 requested review from Copilot and jaydeluca May 9, 2026 02:50

Copilot started reviewing on behalf of SurbhiAgarwal1 May 9, 2026 02:51 View session

Copilot AI reviewed May 9, 2026

View reviewed changes

SurbhiAgarwal1 requested a review from Copilot May 9, 2026 03:12

Copilot started reviewing on behalf of SurbhiAgarwal1 May 9, 2026 03:13 View session

Copilot AI reviewed May 9, 2026

View reviewed changes

jaydeluca reviewed May 11, 2026

View reviewed changes

Merge branch 'main' into feat/242-library-readme-support

762d41f

SurbhiAgarwal1 requested a review from jaydeluca May 11, 2026 16:44

SurbhiAgarwal1 added 6 commits May 11, 2026 22:40

Merge branch 'main' into feat/242-library-readme-support and fix fail…

4863395

…ing tests

fix: resolve merge conflict in fetch-with-cache.ts

c9bb873

fix: resolve merge conflict and apply formatting

186b011

chore: remove untracked generated and temp files from commit

6afc5cb

fix: restore public/data from main (accidentally removed)

5b1a322

fix: restore public/data from main and add prettierignore for data files

556b3b0

SurbhiAgarwal1 and others added 5 commits May 11, 2026 23:56

fix: increase App test timeout for lazy-loaded Suspense and exclude p…

74e232a

…ublic/data from markdown lint

feat: implement library readme support and list pages redesign (open-…

752e936

…telemetry#242)

style: fix formatting and import sorting for CI

f7b8f83

remove generated files

5e2d7bf

Merge branch 'main' into feat/242-library-readme-support

309a487

jaydeluca requested changes May 12, 2026

View reviewed changes

SurbhiAgarwal1 added 7 commits May 12, 2026 22:38

chore: remove unrelated generated files and project artifacts

9ae5cba

chore: ensure all unrelated files match main branch

1fde25b

fix: merge upstream/main and resolve bun.lock conflict

da12718

fix: remove temp files and fix formatting after upstream merge

feb8cfb

fix: apply prettier formatting to root-level files

0f03ea4

fix: remove local description files from repo

9f1c1ad

ci: trigger re-run after format fix

e404018

SurbhiAgarwal1 requested a review from jaydeluca May 13, 2026 06:11

SurbhiAgarwal1 added 12 commits May 14, 2026 11:14

Merge upstream/main and resolve conflicts

213518d

chore: clean up unrelated changes and generated files to resolve PR c…

856e29b

…onflicts

chore: revert unintended registry changes

72d8fdf

chore: remove unrelated files and SNAPSHOT registry data

9aad922

chore: final cleanup of unrelated files

4108acc

chore: properly align unrelated files with upstream/main

e1f654c

chore: revert __init__.py versioning fixes to keep PR focused

110ecb4

chore: restore README files and registry hashes

fa24eaf

Merge upstream/main and resolve conflicts

6ae092c

Add test for loadGlobalConfigurations

224ae67

chore: include remaining local changes after merge

1b5a75a

test: update error message expectations to match new format

4505b8e

Conversation

SurbhiAgarwal1 commented May 7, 2026

Implementation Detail: Library README Support (#242)

1. Registry Discovery & Extraction

2. Metadata Augmentation & Backfilling

3. Database Writing & Publishing

4. Frontend Integration

5. Build System & DX Improvements

6. Verification Summary

Uh oh!

netlify Bot commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for otel-ecosystem-explorer ready!

Uh oh!

linux-foundation-easycla Bot commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jaydeluca left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

jaydeluca left a comment

Choose a reason for hiding this comment

Uh oh!

jaydeluca left a comment

Choose a reason for hiding this comment

Uh oh!

SurbhiAgarwal1 commented May 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

netlify Bot commented May 7, 2026 •

edited

Loading

linux-foundation-easycla Bot commented May 7, 2026 •

edited

Loading